Automatically constructing GUI groups of different granularities constitutes a critical intelligent step towards automating GUI design and implementation tasks. Specifically, in the industrial GUI-to-code process, fragmented layers may decrease the readability and maintainability of generated code, which can be alleviated by grouping semantically consistent fragmented layers in the design prototypes. This study aims to propose a graph-learning-based approach to tackle the fragmented layer grouping problem according to multi-modal information in design prototypes. Our graph learning module consists of self-attention and graph neural network modules. By taking the multimodal fused representation of GUI layers as input, we innovatively group fragmented layers by classifying GUI layers and regressing the bounding boxes of the corresponding GUI components simultaneously. Experiments on two real-world datasets demonstrate that our model achieves state-of-the-art performance. A further user study is also conducted to validate that our approach can assist an intelligent downstream tool in generating more maintainable and readable front-end code.

When translating UI design prototypes to code in industry, automatically generating code from design prototypes can expedite the development of applications and GUI iterations. However, in design prototypes without strict design specifications, UI components may be composed of fragmented elements. Grouping these fragmented elements can greatly improve the readability and maintainability of the generated code. Current methods employ a two-stage strategy that introduces hand-crafted rules to group fragmented elements. Unfortunately, the performance of these methods is not satisfying due to visually overlapped and tiny UI elements. In this study, we propose EGFE, a novel method for automatically End-to-end Grouping Fragmented Elements via UI sequence prediction. To facilitate the UI understanding, we innovatively construct a Transformer encoder to model the relationship between the UI elements with multi-modal representation learning. The evaluation on a dataset of 4606 UI prototypes collected from professional UI designers shows that our method outperforms the state-of-the-art baselines in the precision (by 29.75\%), recall (by 31.07\%), and F1-score (by 30.39\%) at edit distance threshold of 4. In addition, we conduct an empirical study to assess the improvement of the generated front-end code. The results demonstrate the effectiveness of our method on a real software engineering application. Our end-to-end fragmented elements grouping method creates opportunities for improving UI-related software engineering tasks.

Earlier this year Tesla CEO Elon Musk said the future is now. By the middle of 2020, he said at an event for investors, Tesla's autonomous system will have improved to the point where drivers will not have to pay attention to the road. He revealed that Tesla has plans to roll out Level 5 autonomous taxis next year in some parts of the United States, which means they will be capable of driving themselves anywhere on the planet, under all possible conditions, with no limitations. That's compelling, but is it really possible within such a short timeframe? In May, a month after Musk's speech, Consumer Reports said that the new lane-changing feature on Tesla's updated Navigate on Autopilot software lags far behind a human driver's skills.

Although there are designers who claim design is a mysterious activity not amenable to scientific examination, research into design continues Although there are publications by designers on how to design dating back to Roman times, notably by Vitruvius, the nineteenthcentury design thinkers actually began work on articulating design as a process (Durand 1802). However, it was not until the 1960s that major research programs were initiated. These programs were originally founded on the systems view and used concepts from operations research (Jones and Thornley 1963). More recently, information-processing models founded on AI concepts have provided an impetus for renewed research into design in its various aspects (Simon 1969; Coyne et al. 1990). Many foundational ideas in AI are proving to be useful in developing formal models of design as an activity.

This article begins with an elaboration of models of design as a process. It then introduces and describes a knowledge representation schema for design called design prototypes. This schema supports the initiation and continuation of the act of designing. Design prototypes are shown to provide a suitable framework to distinguish routine, innovative, and creative design.

This article begins with an elaboration of models of design as a process. It then introduces and describes a knowledge representation schema for design called design prototypes. This schema supports the initiation and continuation of the act of designing. Design prototypes are shown to provide a suitable framework to distinguish routine, innovative, and creative design.